The subject matter discussed in the background section should not be assumed to be prior art merely as a result of its mention in the background section. Similarly, a problem mentioned in the background section or associated with the subject matter of the background section should not be assumed to have been previously recognized in the prior art. The subject matter in the background section merely represents different approaches, which in and of themselves may also be inventions.
A set of users U interacted with some of the objects o in a universe of objects O in a particular pre-defined way, with varying degrees of success. (u, o), uϵU, oϵO, denotes an interacting (user, object) pair, and an interaction strength wuo is associated with such a pair. A personalized recommendation model recommends, for any user u, objects in O that are similar to the objects o with which the user u has previously interacted. The higher the interaction strength, the more positive the influence the properties of o should have for recommending new objects to the user u. A personalized recommendation system may build a personalized recommendation model Mu, one for each user u, from the historical interaction data of the user u. This historical interaction data of the user u is Du={(o, wuo)|wuo>0}, where wuo>0 is an indicator that user u has interacted with object o. Mu uses the content of object o and the interaction strength of wuo of (u, o) to make recommendations. The personalized recommendation model Mu uses a score function Muscore(o) to assign a score to any object oϵO, reflecting on how similar object o is to user u's data DU on which the model Mu was trained. This score function is used for recommending new objects to user u.
A personalized recommendation system has challenges in evaluating when the personalized recommendation models {Mu} are “good enough” to be used in production and also in evaluating potential improvements to the personalized recommendation models {Mu}. Such evaluations seem to require human identification of objects that form good recommendations for a sufficiently large set of users, and require human identification of objects that form bad recommendations for a sufficiently large set of users. Such human identification, which might be very useful, tends to be very laborious and error-prone. Accordingly, it is desirable to provide techniques for evaluating personalized recommendation models that do not require such human identification.